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1.  INTRODUCTION 


Hackn-found 


For  a number  of  years,  the  Air  Force  Human  Resources 
Laboratory  has  sponsored  research  into  methods  for-  measuring 
and  improving  tire  comprehensibility  of  textual  materials  used  dur- 
ing Air  Force  technical  training.  The  long  range  goal  of  this  re- 
search has  been  to  develop  methods  for  reliably  measuring  and  in- 
creasing the  comprehensibility  of  textual  materials  and  thereby 
allow  a reduction  in  training  time  and  costs,  as  well  as  an  in  -roast 
in  training  effectiveness. 

A number  of  efforts  toward  this  goal  have  been  completed 
to  date.  Williams,  Siegel,  and  Burkett  (1974)  reviewed  the  methods 
for  predicting  comprehensibility  of  text  which  were  developed 
through  the  year  1972.  They  pointed  out  that  research  activity  in- 
to language  behavior  relevant  to  reading  comprehension  was  grow- 
ing, and  that  no  theories  or  hypotheses  useful  in  the  prediction  of 
textual  comprehensibility  were  available  until  very  recent  years. 

Siegel  and  Bergman  (1974)  developed  and  tested  a set  of  var- 
iables related  to  readability  which  were  based  on  selected  intellec  - 
tive activities  as  described  in  the  Structure-of-Intelleet  model  of 
Guilford  (1966,  196  7).  Lambert  and  Siegel  (1  974)  followed  similar 
procedures  in  developing  and  testing  a set  of  variables  based  on 
constructs  described  in  the  psycholinguistic  literature.  Siegel, 
Williams,  Lapinsky,  Warms,  Wolf,  Groff,  and  Burkett  0 976  ) investi 
gated  these  Structure-of-Intcllect  and  psycholinguistic  variables 
and  presented  a technique  for  measuring  the  variables  by  computer. 

In  addition,  two  other  technical  reports  have  been  prepared. 
In  one  (Siegel,  Federman,  & Burkett,  1974),  techniques  arc  pre- 
sented, in  nontechnical  language,  for-  predicting  text  readability 
and  for  experimentally  measuring  text  readability.  The-  second 
report  (Siegel,  Lambert,  & Burkett,  1974)  presents  a series  of 
concepts  the  implementation  of  which  would  tend  to  improve  the 
comprehensibility  of  technical  writing. 

The  research  in  this  report  extends  the  prior  studies  re- 
lated to  Structure-of-Intcllect  and  psycholinguistically  oriented 
vari  ablos. 
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Readability  Indices 


Williams  ot  al.  (1!'74)  described  ll!  roadability  formulas 
developed  prior  to  1 ? ‘ 7 ' i . Cicnorally,  tlie  procedure  followed  in 
developing  these  formulas  involved  computing  a multiple  linear 
regression  equation  which  related  measurable  characteristics 
of  text  to  comprehensibility,  defined  as  a comprehension  test 
score  [such  as  cloze  test  score  (Taylor,  1 f ' 5 . 1 ) J or  judged  difficul- 
ty. 


The  factors  included  in  most,  if  not  all,  readability  equa- 
tions have  involved  measures  taken  within  sentences  only.  \o 
readability  equation  exists,  to  our  knowledge,  which  considers 
factors  distributed  across  sentences,  such  as  ideational  density, 
organization,  obscurity  of  expression,  etc.  1'hese,  however, 
are  the  points  which  are  emphasized  in  various  guides  for  read- 
able writing. 

The  factors  which  are  considered  in  r<  adability  equations, 
in  addition,  have  been  chosen  because  the\  .ire  easily  measured. 
They  did  not  stem  from  theories  or  hypotlw-t  regarding  language 
processing.  Accordingly,  it  can  be  held  that  they  lack  construct 
validity.  One  would  expect  that  readabilit  measures  based  on 
theory  or  hypotheses  concerning  language  processing  would  be 
more  useful  and  possess  greater  diagnostic  value  than  measures 
based  on  molecular  characteristics  of  text  which,  if  manipulated 
in  isolation,  have  little  effect  on  comprehensibility.  For  example, 
the  writing  of  Gertrude  Stein  is  very  difficult  to  read,  despite  the 
short  sentences  and  simple  vocabulary  which  characterize  her  work. 
The  two  most  popular  readability  formulas,  Flesch  "reading  ease" 
and  the  Dale-Chall  formula,  rate  samples  of  Stein's  work  as  very 
easy  reading  (Taylor,  1953). 

Also,  the  developers  of  prior  readability  predictive  equa- 
tions have  frequently  pointed  out  that  their  indices  only  measure 
readability.  They  tell  the  author  little  about  how  a given  text  may 
be  modified  to  increase  its  readability.  In  fact,  developers  of  these 
indices  have  warned  against  employing  their  formulas  in  reverse 
to  increase  readability. 

This  fact  along  with  the  inability  of  readability  equations  to 
accurately  measure  atypical  text  suggest  that  they  do  not  address 
the  factors  or  variables  which  are  actually  involved  in  human 


processing  of  written  language.  Rather,  such  factors  only  address 
correlates  of  the  "true"  factors.  Bormuth  (1969)  argued  that  only 
sentence  structure  and  syntactic  complexity  may  determine  compre- 
hension difficulty.  Modification  of  word  length  and  sentence  length 
may  influence  comprehension  only  to  the  degree  that  they  affect  the 
syntactical  or  phonemic  structure  of  messages. 

In  order  to  extend  the  constructional  scope  of  textual  com- 
prehensibility measurement  and  incorporate  a diagnostic  capability, 
the  Air  Force  Human  Resources  Laboratory  has  sponsored  research 
to  define,  quantify,  and  verify  comprehensibility  measures  which 
are  based  on  concepts  drawn  from  both  the  psycholinguistic  and  the 
intellective  structure  literature. 


Psy  choli  ngui  s t ic  ally  Oriented 
Comprehensibility  Measures 


In  an  attempt  to  isolate  textual  comprehensibility  measures 
based  on  or  oriented  towards  language  processing  constructs, 
Lambert  and  Siegel  (1  974)  surveyed  the  literature  to  isolate  psy- 
cholinguistically  oriented  variables  which  might  influence  reading 
comprehension.  They  identified  a number  of  such  variables: 

Yngve  depth,  morpheme  depth,  transformational  complexity,  cen- 
ter embeddedness,  left  branching,  right  branching,  and  comple- 
ment deletion. 

As  pointed  out  by  Bormuth  (1966),  until  very  recent  years 
no  theoretical  base  was  available  from  which  to  generate  testable 
hypotheses  relating  to  readability.  Modern  linguistic  and  psycho- 
linguistic  research  activities  have  begun  to  fill  this  void.  But  due 
to  the  apparent  effectiveness  of  the  simple,  structural  variables 
and  the  complexity  of  the  variables  contained  in  new  language  the- 
ories, researchers  still  seemed  to  lean  toward  the  structural  vari- 
able’s (e.  g,,  Coleman  k Liau,  1975).  Only  research-oriented  for- 
mulas, such  as  those  of  Bormuth  (196  9),  seemed  to  have  employed 
variables  based  on  theory.  However,  the  variables  in  Bormuth's 
formulas  are  difficult  to  measure  and  rely  on  sentence  analysis  by 
skilled  persons. 
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The  current  effort  to  develop  a readability  equation  for  ap- 
plied use  was  undertaken  with  the  assumption  that  the  variables 
could  be  measured  bv  computer.  We  assume  that  programs  can 
be  written  which  will  perform  certain  sophisticated  language  anal- 
yses which  are  too  t i the  consuming  and  difficult  for  routine  manu- 
al performance,  and  Siegel  et  al.  (1976)  provides  a detailed  de- 
scription of  a computer  program  capable  of  measuring  the  vari- 
ables described  below. 

Yngve  Depth  (YU) 

Yngve  (I960)  developed  a model  of  sentence  production  on 
the  basis  of  conjecture  that  a person  produces  sentences  by  gener- 
ating a'  "sentence  structure  tree"  in  a top-to-bottom,  left-to-right 
direction.  According  to  this  model,  at  any  given  time  a speaker 
has  produced  only  that  portion  of  the  left-hand  side  of  the  tree  nec- 
essary to  produce  the  word  spoken.  As  the  speaker  works  down 
the  tree,  he  produces  both  branches  of  a node,  but  he  must  store 
the  right  branch  in  memory  while  he  is  expanding  the  left  branch. 
Yngve  depth  is,  thus,  a measure  of  the  memory  demand  placed  on 
the  person  reading  orauding  the  sentence.  Bormuth  (I960)  found 
that  sentence  depth  was  correlated  with  the  difficulty  ot  a passage. 
Martin  and  Roberts  (1966)  held  sentence  length  constant  and  varied 
the  Yngve  depth.  They  found  that  sentences  of  lesser  complexity 
were  recalled  significantly  more  frequently  than  sentences  of  great- 
er structural  complexity.  The  finding  that  mean  linguistic  depth  is 
a strong  predictor  of  sentence  comprehensibility  has  been  confirm- 
ed by  Wang  (1970),  i.ambert,  and  Siegel  (1974),  and  Siegel  et  al. 
(1976). 

\lorpheme  Depth  (Ml)) 

A morpheme  is  the  meaning  carrying  unit  of  language,  and 
does  not  always  correspond  to  the  syllable.  For  example,  the 
word  "flower"  is  one  morpheme.  Bormuth  (1969)  speculated  that 
the  comprehensibility  of  an  individual  word  may  depend  on  how 
manv  morphemes  are  'buried'  within  it.  ( onsider  the  word 

un/happi/ness 

un  morpheme-  denoting  "not" 

haj  | i mar;  ■ m denoting  a state  of  mood 
no.  morph* -me  denoting  a condition  or 

qual ity 
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A person  reading  this  word  must  have  knowledge  of  the  meaning  of 
all  three  morphemes  to  comprehend  the  word. 

Lambert  and  Siegel  ( 1 1174)  and  Siegel  et  al.  (1976)  found  the 
mean  number  of  morphemes  per  word  to  be  related  to  comprehen- 
sibility. 

Transformational  Complexity  (TC ' ) 


According  to  theories  of  transformational  grammar,  sen- 
tences of  any  type  or  level  of  complexity  are  produced,  or  inter- 
preted, through  transformations  relative  to  simple,  active  "ker- 
nels. " Interpretation  of  passive,  negative,  or  passive  negative 
sentences  or  independent  clauses  requires  successively  more,  or 
more  elaborate,  transformations  from  the  basic  active  kernel.  The 
TC  of  a sentence  is  a measure  of  the  number  of  transformations 
from  its  kernel.  Lambert  and  Siegel  (1974)  found  the  relative  accu- 
racy of  interpretation  of  sentences  of  these  four  classes  of  trans- 
formations to  decline  in  the  order  described  above. 

Center  Embedding  (CE) 

If  phrases  or  clauses  appear  between  the  subject  and  the 
predicate  of  a sentence,  such  phrases  or  clauses  are  said  to  be 
center  embedded.  Schwartz  et  al.  (1970)  demonstrated  that  inclu- 
sion of  center  embedded  material  decreased  comprehensibility. 

The  results  of  Lambert  and  Siegel  (1.974)  supported  the  findings  of 
Schwartz  et  al.  However,  Siegel  et  al.  (1976)  found  that  center  em- 
bedded materials  were  associated  with  higher  cloze  scores.  They 
suggested  test  method  bias  as  a possible  cause  of  the  inconsistent 
findings  across  studies. 

Left  Branching  (LB)  and  Right  Branching  (RB) 

Schwartz  et  al.  (1970)  also  found  that  addition  of  phrases 
or  clauses  to  the  left  of  the  subject  decreased  comprehensibility. 
Addition  of  similar  material  following  the  sentence  predicate  did 
not  degrade  comprehensibility.  Lambert  and  Siegel  (1974)  tested 
these  variables  and  obtained  mixed  results.  Siegel  et  al.  (1976) 
found  both  the  left  branching  and  the  right  branching  variables  to 
affect  cloze  score,  although  the  effect  of  right  branching  was  not 
in  the  anticipated  direction. 


13 


— ■ 


Complement  Deletion  (DC) 

It  has  been  hypothesized  that  certain  surface  structures  of 
language,  such  as  the  complement  "that"  in  ' He  said  that  I should 
go,  11  may  serve  to  mark  the  deep  structure  of  a sentence,  and  so 
contribute  to  its  comprehensibility.  Hakes  (1 972 ) demonstrated 
this  effect.  Lambert  and  Siegel's  (1974)  results  disagreed  with 
Hakes'  and  found  a significant  effect  in  the  opposite  direction. 
Siegel  et  al.  (1976)  corroborated  the  findings  of  I.ambert  and 
Siegel. 


Intellective  Theory  Oriented 
Comprehensibility  Measures 


Guilford  and  his  associates  (1950,  1954,  1964,  1966,  1967) 
developed  a three-factor  taxonomy  of  human  mental  activity.  Based 
on  factor  analytic  procedures,  120  nonoverlapping  intellective  ac- 
tivities have  been  described.  The  three  factors  isolated  by  Guilford 
include:  (1)  "contents,  " indicative  of  the  form  in  which  information 
mav  be  presented,  (2)  "operations,  ' describing  the  types  of  proc- 
essing applied  to  the  information,  and  (3)  "products,  ' which  describe 
the-  forms  in  which  the  output  of  the  operation  may  occur.  Within 
contents,  four  categories  exist;  figural,  symbolic,  semantic,  and 
behavioral.  Five  operations  are  identified  (cognition,  memory, 
convergent  production,  divergent  production,  and  evaluation)  as 
well  as  six  categories  of  output  (units,  classes,  relations,  systems, 
transformations,  and  implications).  Each  combination  of  one  con- 
tent, one  operation,  and  one  product  represents  one  unique  class 
of  intellective  function.  . The  classes,  based  as  they  arc  on  combina- 
tions of  categories  within  three  orthogonal  factors,  may  be  repre- 
sented as  a cube  composed  of  120  cells.  A representation  of  the 
Guilford  Structure-of-Intellcct  (SI)  model  is  presented  as  Figure  1. 
Guilford  and  his  colleagues  have  identified  examples  of  perform- 
ance and  tests  for  the  majority  of  the  120  cells. 

Siegel  and  Bergman  (1974)  hypothesized  that  textual  materi- 
als which  require  a high  level  of  SI  ability  would  lie  less  compre- 
hensible than  would  texts  requiring  the  Same  SI  ability  at  a lower 
level.  This  work  examined  various  Guilford  categories  and  se- 
lected eight  which  seemed  most  relevant  to  the  readability/ com  - 
prehensibility  problem  in  the  Air  Force  technical  training  context. 
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OPERATION 

Evaluation 


Convergent  production 
Divergent  production 


Figure  1.  Guilford's  Structure-of-Intellect  model. 

(Guilford  S Hoepfner,  1971) 
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Siegel  ami  liergman  postulated  these  Guilford  abilities  to 
reproseut  an  intervening  variable  between  the  surface  structure'  of 
a text  (the'  structure  as  the  message  is  presented  on  a page)  and  the' 
deep  structure  (the'  structure  of  the'  message  after  transformation  to 
simplest  form  is  completed).  This  intervening  variable,  called  in- 
tellective loael,  has  to  do  with  the  intellective  processes  (type  and 
amount)  required  for  converting  the  coded  language  into  a meaning- 
ful message. 

Cognition  of  Semantic  Units  (GiUtJ) 

Guilford  (1967)  defined  cognition  of  semantic  units  as  a vo- 
cabulary factor,  stating  it  is  most  directly  measured  by  vocabulary 
tests.  Here,  t'MU  .in  the  context  of  prose  reading  is  held  to  involve 
the  extent  to  which  the  text  requires  the  reader  to  recognize  a diver- 
sity of  \Vord  forms.  Measures  of  vocabulary  diversity  are  common- 
ly found  in  existing  readability  formulas,  and  Siegel  and  Bergman 
(1974)  found  a significant  effect  of  vocabulary  diversity  on  compre- 
hensibility. The  C MU  variable,  equal  to  the  type-token  ratio  sub- 
tracted from  one,  was  found  by  Siegel  et  al.  (1076)  to  signiticantly 
affect  cloze  score  for  both  low  and  high  ability  readers. 

Cognition  of  Semantic  Relations  (C.MR) 

Cognition  of  semantic  relations  was  defined  by  Guilford  (1967) 
as  the  ability  to  recognize  a relationship  between  items  or  words  and 
analogy  and  word  linkage  tests  were  used  to  measure  this  ability. 
Siegel  and  Bergman  (1974)  demonstrated  that  incomplete  analogies 
and  unclear  word  linkages  degrade  comprehension,  and  Siegel  et  al. 
(1976)  developed  a CM  It  comprehensibility  measure  which  attempted 
to  assess  unclear  linkages  through  the  number  of  shared  nouns  (nouns  ap- 
pearing in  adjacent  sentences)  and  number  of  pronouns.  Siegel  et  al. 
(1  976)  verified  that  such  a measure  significantly  influenced  cloze 
score  for  high-  and  low-ability  readers. 

Memory  for  Semantic  Units  (MMlO 

According  to  Guilford  (1967),  memory  for  ideas  is  most  di- 
rectly addressed  by  memory  for  semantic  units.  Siegel  and  Bergman 
(1  974)  demonstrated  that  replication  of  facts  increased  comprehen- 
sibility. Assuming  that  the  new  ideas  will  be  presented  using  new 
nouns,"  an  analogous  measure  seemed  to  be  the  number  of  different 
nouns  appearing  in  a passage  divided  by  the  number  of  words  in  the 
passage;  this  measure  seemed  to  serve  as  an  index  of  memory  but 
den.  Siegel  et  al.  found  that  this  M MU  measure  affected  compre- 
heu  ion  in  the  hypothesized  direction. 
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Evaluation  of  Symbolic  Implications  (ESI) 


Guilford  (1967)  used  abbreviations  tests  to  measure  evalu- 
ation of  symbolic  units.  A corresponding  readability  measure  was 
thought  to  involve  the  frequency  of  occurrence  of  abbreviations  in 
passages  of  running  text.  Siegel  and  Bergman  (1974)  and  Siegel  et 
al.  (1976)  demonstrated  that  frequency  of  occurrence  of  abbrevia- 
tions influenced  comprehension. 

Convergent  Production  of  Semantic  Implications  (\MI) 

Guilford  and  Hoepfner  (1971)  used  syllogisms,  attribute  list- 
ing, missing  links,  and  sequential  association  tests  to  measure  con- 
vergent production  of  semantic  implications.  Beading  material  load- 
ed in  convergent  production  requires  the  reader  to  perform  syllogis- 
tic reasoning  tasks,  and  material  which  does  not  require  this  ability 
would  complete  the  syllogism  for  the  reader.  Increase  of  the  con- 
vergent production  of  semantic  implications  load  in  a text  should  de- 
crease comprehensibility.  Siegel  and  Bergman  (1974)  found  that 
textual  material  which  imposed  syllogistic  reasoning  demands  on 
the  reader  was  less  comprehensible  than  that  not  demanding  such 
reasoning. 

Identification  of  the  number  of  syllogisms  in  a text  is  quite 
difficult.  Accordingly,  a measure  was  developed  in  which  the  num- 
ber of  words  of  a passage  is  divided  by  the  total  of  the  dictionary- 
listed  numbers  of  parts  of  speech  of  the  words  of  the  textual  sample, 
['his  was  considered  to  be  an  index  of  the  number  of  possible  parses 
of  a sentence  and  thus  analogous  to  a measure  of  syllogistic  reason- 
ing. Siegel  et  al.  (19  7b)  found  this  measure  to  be  related  to  compre- 
hensibility, although  not  in  the  predicted  direction. 

Divergent  Production  of  Semantic  1 nits  (l)Alt  ) 

According  to  Guilford  (1967),  divergent  production  of  seman- 
tic units  involves  the  ability  to  enumerate  class  members  given  cer- 
tain class  properties.  With  regard  to  the  readability  of  training 
texts,  divergent  production  <5 f semantic  units  would  require  the  read- 
er to  enumerate  class  members  on  his  own  rather  than  have  the  class 
member  supplied  by  the  reading  selection.  Siegel  and  Bergman  (1974) 
demonstrated  a relationship  between  the  presentation  of  examples  and 
comprehension,  and  the  results  of  Siegel  et  al.  (1976  ) confirmed  these 
findings. 
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C ' o ny ergent  Production  of  Semantic  Systems  (\  US) 


Guilford  ( 1 f ' 6 7 ) relates  convergent  production  of  semantic 
systems  to  skill  in  ordering.  Siegel  and  Bergman  11974)  demon- 
strated that  passages  containing  mnemonic  devices  and  similar 
memory  aids  were  more  readily  comprehended  by  readers  than 
were  similar  passages  lacking  these  aids.  The  \ MS  measure  is 
defined  as  the  number  of  such  aids  in  a block  of  text  arbitraril\ 
divided  by  four. 


Objectives  of  B r e s e n t \V  o r k 


As  described  above,  a series  of  studies  into  the  potential 
usefulness  of  psycholinguistic  and  intellective  oriented  variables 
for  measuring  and  increasing  the  comprehensibility  of  textual  ma- 
terials has  already  been  completed.  These  studies  established 
methods  for  measuring  these  variables  and  evaluated  the  measures. 
Additionally,  these  studies  provided  a function  specification  for  a 
computer  program  for  calculating  these  measures. 

The  goals  of  the  present  work  were  to  extend  the  prior  work 

and  to; 


1,  develop  norms  of  occurrence  of  the  psycho - 
linguistically  and  the  SI  oriented  comprehen- 
sibility variables  previously  developed  and 
desc  ribed; 

2.  further  verify  the  measures  and  develop  the 
best  linear  combination  of  the  variables  for 
predicting  the  comprehensibility  of  Air  force 
training  materials. 
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The  psycholinguistic  and  SI  oriented  comprehensibility  meas- 
ures which  were  included  in  prior  studies  (Chapter  1)  and  in  the  func- 
tional computer  implementation  description  are  presented  and  defined 
in  Tables  1 and  2. 

Normative  data  were  desired  describing  the  occurrence  of  the 
listed  psycholinguistic  and  SI  variables  in  four  types  of  Air  Force 
publications:  (1)  study  guides  employed  in  formal  classroom  training, 
(2)  manuals  and  regulations  of  the  type  used  for  field  and  occasional 
classroom  reference,  (3)  career  development  course  (('IX')  texts 
(self-study  materials  taken  by  enlisted  personnel  to  meet  a portion 
of  the  requirements  for  skill  upgrading),  and  (4)  technical  manuals 
(publications  presenting  the  specific  methods  and  procedures  to  be 
followed  on  the  job  and  related  information).  One  hundred  fifty 
nonsequential  pages  from  each  category  of  document  were  provided 
for  this  purpose  by  the  Air  Force  Human  Resources  Laboratory. 

The  range  of  subjects  represented  by  the  provided  pages  is  indicated 
in  Table  3. 

From  each  of  the  Table  3 categories,  50  blocks  of  text  were 
selected  to  yield  normative  data.  In  order  to  select  each  desired 
sample  of  50  blocks,  the  number  of  column  pages  of  running  prose 
in  each  sampled  category  was  determined.  [In  instances  in  which 
there  were  two  columns  of  text  on  a single  page,  as  in  ( DCs,  each 
column  was  considered  as  a separate  page.  ] The  total  of  the  col- 
umn pages  was  then  divided  by  50.  The  number  so  obtained  repre- 
sented the  interval  between  beginnings  of  the  passage  blocks  se- 
lected. The  starting  point  of  a block  was  adjusted  forward  or  back- 
ward, whichever  allowed  the  least  displacement,  so  that  a major 
change  of  topic  did  not  occur  within  the  finally  chosen  block. 

Fach  block  began  at  the  beginning  of  a sentence  and  ended 
at  the  first  sentence  end  after  the  99th  successive  word.  Selected 
passages  were,  thus,  approximately  100  words  in  length.  Only  sen- 
tences of  running  prose  were  included  in  the  selected  samples. 

Word  counting  was  suspended  when  paragraph  headings,  lists  pre- 
sented in  outline  form,  and  the  like  were  encountered.  Samples 
were  not  strictly  limited  to  a particular  number  of  words  since 
several  measures  were  derived  from  characteristics  of  whole  sen- 
tences (e.  g.  , ('.MR,  l)MF,  YD)  or  employed  number  of  sentences  as 
a base. 
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Table  1 


Summary  of  Comprehensibility  Measures 
Formula  Abbreviation  Name 


Structu  re -of -Intellect 


, MDWB 

TNWB 

CHU 

Cognition  of  Semantic  Unite 

NS  MB  \ 

TNSB  ~ + N0RBj  /TNWB 

CMR 

1 logn  i t Lon  of  S<  :mant  Lc  Relations 

t NDNB 
TNWB 

MHU 

Mi  m >ry  1 or  Semant ic  Un i ts 

. NSWB 

TNWB 

ESI 

Eval tat  ion  of  Symbolic  Imj li  ition 

TNWB 

TPSB 

::m  i 

mvergent  Production  of  Semantic 
Implications 

TNKB 

I 

Divergent  Prc  luction  of  Semantic 

Units 

TMAB 

4 

NMS 

Convergent  Production  of  Semantic 
Systems 

Psycholinguistic 

- 

I'NWB 

iN.'B  • >:  VijS 

w 

YD 

Yngve  depth 

itwb/tnmb 

HD 

\ 

Morpheme  depth 

j/'J  'i. . :i 

TC 

Transformat ional  complexity 

; ; ; i 1 1 1 ' ( ; 

CE 

Center  embedding 

TNGB 

LB 

Left  branching 

['  * ■ 1 

RB 

Right:  branching 

1 / I 

DC 

Deleted  complements 

Tabic;  2 


NDWB 

l'NWB 

NSNB 

NOKB 

NDNB 

NSWB 

NPPB 

TNMB 

TNEB 

I'NSB 

NNPS 

NCRS 

DCS 

YDW 

NCLS 

YDS 

TCS 

TPSB 

TNAB 


Symbology  for  Table  1 bo rnnilac 

Number  o 1 difierent  wor-‘  in  a text  block 
Total  number  oi  word:-,  in  a text  block 
ti  : er  oi  sh  tre  1 n m , mi  >f  n<  ms  in  ad  ja  • ni  :ent<  nee 
Num  oi  reference  (nui  f pron  in  ) in  a t( 1 bio  ■ 

Number  of  different  noun.-  in  a text  block 
Nurol  er  oi  abbrev iated  oi  symbolic  words  in  a t ■ x - b 1 < :) 

Number  of  potential  parse:;  per  sentence 
Total  number  of  morpheme.'.:  per  block 
Total  number  of  elucidation:,  per  block 
Total  number  of  sentence..;  pet  block 

Number  of  noun  phrases  to  the  right  oi  the  . ul  ’•  t verl  in  a 
sentence 

Number  of  modifying  clauses  on  the  right  of  the  object  noun 

phrase  of  a sentence 

Deleted  complement  in  a sentence 

Sum  of  all  digits  on  parse  paths  to  each  word  of  block 
Number  of  chained  modifying  clauses  on  the:  left  of  the  sub  joe 
noun  of  a sentence 
Yngve  depth  of  sentence 

Transformational  complexity  value  for  a sentence 
Total  number  of  parts  of  speech  in  all  words  oi  a bl< 

Total  number  of  aids  per  block 


21 


» r»  .<  »/<  ,« 


P tilde 


S'uii't  ■ I)'  I \nul.Y/.rd  Co r J Jevcli)|>nif  ill  of  Norm 

;•  •:  > ( 1 ) . ■ I ti>Iy  Ou  i m ■ 

' . ■ . Ai  i I i ■:  ( ■>  i i n I I y or  C>  a N m. 

t 1 1 1 1 hill  ; ! >1 1 i ■ S[  i:f  ! . 1 1 i a t 

: ii-  i Spee  i i 1 i st 

1 M . ■ l • ,u  W<  at. on  . lal ist 

■ ! • ; , i a tp  a .'if!  n,-'.  I!  pii  piv  ‘Ii  i Spec  i 1 1 1 : t 

ii  ■ Lpli  { M > ir  Self 
i ; K hi.  i ' "!•  -n  L : : " 'c in  li  .t 


lib:  i y ( ? ) . M inua  I ■ , <in>l  Riaju  I a f ion  a 

1 1 mua  I N 1 . Con  I an  1 

AIM  / 1 U •.»!'  :•  ;,a  ly  Manual 

. ■ ilniiu'  : ; un.lar  I , Proimt  Ion  Pitaes:: 

Kx  !’  imii  'oil.  Study  Guide 

klent  revent i "a  1 

AIM  ' . i. • li.’-  'cjf.  Train  in.:' 

l,al  Equipi 

. ■ ■ ' . a In:  1 i , (or  Pi'!  : i : : i on  Photographic 

’ ns'  Laboratories 

iqui  n ' ; t I ■ n 

, 5?-  ■ ■ i< 


; i/e  lopmoril  irs  lexf 

C<  u i ■ Ai  f f ore  ■ 3 p ■ : - i a | I y 

I ' - " . ■ ■ ! i i r.  :ri  i r ;■  i ’ ! n ‘ I ; ••rial  i:;t 

CIKV31  Still  Photo;'  t a,  i Ppeui  ilir.t 

TK.MvJSn  Cent  itiUO-l  ••  Pi. am.  :..••••  . ill:  r'i  ,•[  iuliu.t 

-3:  ' H 3 ‘ , i Nuclf  n Weu;  . f',j  tv  i i'  ist 

164  tory  Manaj  t pecialisl 

' ' ’ ' idli 


Calf-j  Ty  (4) . I • I in  P 1 1 M-jiiu  1 1 u 

inua  I Mo.  ] i I |n 

r.O. IF-4C-2-1  Kaintenai  i , >y  terns,  U \ 

Seritu  i -4'"  A i ivruf  t 

t.  1 P-  ’ 1 1 A-L'-ll  1 ; : i ' : ' : i . it  . 1 I : ' . ' ■ ' i:  ■ , ill'  , USA:' 

Serin..  !'-l  1 1A  A iraaf  t 

r.O.  ' -fit  1 $ JB-111A  Shoj  ystern 

. . . : . I - ! • . ■ • P f'll.’.ht  \‘>nt  i vp  1 i'.y:;t  nn: ; , 

USA  I I 1111;  Ail  raft 

7.0.''  ' p i • i t r • In’.'  aii'f  Touting  Praclicu  , 

. . t ' . : ' , V ' o T<  ",  t.  . t a ‘ i on  T;,u 

am/ a:  ' 

. . ’ . ’ • : i 1 • v a:.  Au,  ; it  • • •:;  o'  rar.  ' I tvr  . 

7 . ’ . i • /.’!**  ! "■  ' : 1 i . 1 1 ii;: 


I *.  •/'  ....  r»- 


Analysis  of  Text  Samples 


Analysis  of  the  selected  textual  passages  was  performed  by 
a team  of  persons  who  are  highly  familiar  w ith  grammatical  and  re- 
lated language  constructs.  1 he  analytic  team  members  were  upper- 
level  college  undergraduates  and  graduate  students  majoring  in  litt  r- 
ature,  foreign  languages,  and  linguistics.  They  analyzed  the  se- 
lected text  passages  according  to  a strictly  specified  set  of  rules 
and  descriptions  of  the  various  measures.  These  rules  were  con- 
tained in  a specially  prepared  "analyst's  manual.  " Prior  to  per- 
forming any  analyses,  the  analysts  were  given  three  days  of  formal 
training  and  practice  in  deriving  the  various  measures. 

In  order  to  determine  whether  the  analysts  were  computing 
the  SI  and  psycholinguistic  variables  with  a reasonable  degree  of 
accuracy,  the  word  counts  involved  in  the  calculations  of  six  vari- 
ables were  selected  for  intensive  study.  Analyst  data  from  two 
passages  were  inspected  to  determine  consistency.  On  three  of 
the  variables  (TP,  DP,  and  ESP)  there  was  absolutely  no  variation 
between  analysts  on  either  of  the  passages,  though  there  was  vari- 
ance across  passages.  That  is,  each  analyst  arrived  at  the  same 
score  for  each  variable  within  a passage.  On  the  remaining  three 
variables  (AID,  HI  All  , and  (Ail1)  there  was  some  variation  both  be- 
tween analysts  and  between  passages.  These  data  were  subjected 
to  analysis  of  variance  following  the  procedure  given  in  Winer  (1971, 
section  4.  5).  The  reliability  of  six  judges  (analysts)  is  approximated 
by  the  expression  1 - AlSwithin/  AlSbetween.  where  VISwithin  refers  to 
the  variation  averaged  across  judges  and  AlSbetween  is  the  variation 
between  passages.  Unbiased  estimates  of  these  reliabilities  were 
. 84,  .91,  and  . 98  respectively.  Single  judge  reliabilities  were 
lower--.  40  (AID),  . 63(M\Il  ),  and  . 95(P  All' ),  respectively. 

Hating  of  the  remaining  measures  was  performed  by  two 
or  three  members  of  the  original  group.  Within  the  smaller  group, 
a policy  was  initiated  according  to  which  all  work  was  checked  by 
at  least  one  person  other  than  the  one  who  took  the  original  meas- 
ure. In  this  wav,  the  manner  of  dealing  with  measures  in  which 
some  variability  was  possible  (e.  g.  , Yngve  depth,  YD,  and  conver- 
gent production  of  semantic  implications,  \ All ) was  made  consistent, 
and  the  accuracy  of  the  final  measures  was  controlled. 
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Results  of  Text  Analysis 


On  completion  of  the  analysis  of  a category  of  text  material, 
for  each  measure,  the  50  obtained  values  were  ordered  by  magni- 
tude, and  the  0 deciles  were  identified.  The  first  decile  is  equal  to 
the  mean  of  the  fifth  and  sixth  values  from  the  bottom  of  the  ordered 
list,  the  second  decile  is  equal  to  the  mean  of  the  tenth  and  < leventh 
values,  and  so  on.  These  values  are  presented,  for  each  category 
of  text,  in  Appendix  A to  this  report.  In  these  Appendix  A tables, 
the  mean  of  the  four  category  values  at  each  decile  level  is  shown 
as  an  "overall"  value.  For  each  decile  on  each  measure,  the  set 
of  overall  decile  values  may  be  taken  as  the  best  description  of  the 
occurrence  of  the  measured  variables  in  Air  Force  technical  liter- 
ature in  general. 

Measurement  of  Variables  in  Non  Air  Force  Literature 


Data  describing  the  occurrence  of  the  psycholinguistic  and 
SI  oriented  variables  in  common  literature  were  calculated  in  order 
to  compare  the  characteristics  of  Air  Force  and  civilian  writing. 

For  these  purposes,  a sample  of  passages  was  sought  which  would 
reflect  the  writing  styles  characteristic  of  popular  and  technical 
writing.  Three  passages  of  approximately  100  words  were  random- 
ly selected  from:  (1)  each  of  two  popular  novels,  (2)  a popular  peri- 
odical, (2)  a junior  high  school  mathematics  text,  (4)  a high  school 
physics  text,  and  (5)  a technical  journal  containing  articles  by  con- 
temporary scientists,  l'ach  passage  began  at  the  beginning  of  a sen- 
tence and  ended  at  the  first  sentence  end  following  the  99th  succes- 
sive word  of  running  text.  This,  as  well  as  all  other  details  of 
sample  selection  procedure,  paralleled  the  procedures  followed  in 
the  selection  of  the  Air  Force  text  samples.  Table  4 presents  the 
sources  of  this  sample,  which  is  referred  to  as  the  "general"  sam- 
ple in  subsequent  sections  of  this  report. 

Results  of  Analysis  of  General  Sample 

As  in  the  previous  analysis,  the  obtained  comprehensibility 
values  were  rank  ordered  by  magnitude.  A table  of  values,  based 
on  these  111  samples,  is  presented  as  Table  A-15  of  Appendix  A. 


Table-  4 
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Sources  of  Samples  of  General  Writing 

Steinbeck,  J.  tannery  Row.  Ni  w Y .•  • : Vi)  ing  Pr<  , 1945. 

Hemingway,  C.  lor  Whom  the  Bell  Tolls.  New  York:  . 

Scribner's  Sons,  1940. 

Reader's  Digest.  January  1974. 

uundlach,  B.  , Buffie,  E.  .,  Denny,  R.R.  , f,  Kempf,  A.i'.  Junior 
High  School  Mathematics  8.  River  Forest,  111.:  Laidlaw 
Brothers,  1968. 

White,  H.E.  Physics,  An  Exact  Science.  New  York:  Van  Nostrand. 
1959.  ‘ 

Willis,  M . : .,  Shen,  M. , 6 dray,  K.J.  Canadian  Journal  of  hemi- 
| cal  Engineering,  1974,  52,  331-337. 


Discussion 


The  cumulative  percentile  values  obtained  for  each  variable 
through  the  measurement  of  the  .Air  Force  and  the  general  sample 
variables  are  plotted  in  Figures  2 through  15.  While  the  data  points 
describing  the  occurrence  of  the  variables  in  the  general  text  sample 
are  based  on  a relatively  small  sample  compared  with  that  for  the 
Air  F’orce  material,  the  comparison  of  the  corresponding  distribu- 
tions is  of  interest.  In  reviewing  these  figures,  it  should  be  noted 
. that  the  scales  of  the  abscissas  vary. 

Structure-of-Intellect  Oriented  Variables 
f 

i Figure  2 indicates  a consistently  higher  (p  < . 01,  sign  test) 

(more  comprehensible)  level  of  CMU  for  the  Air  Force  materials 
as  compared  with  the  general  literature.  The  relative  restriction 
of  vocabulary  diversity  in  the  Air  Force  samples  may  reflect  the 
technical  nature  of  the  materials  sampled.  The  subject  of  technical 
material  changes  rarely.  Accordingly,  new  sets  of  nouns  may  be 
less  often  introduced  than  in  more  popular  materials. 
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FIGURE  2.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES  FIGURE  3.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES 

OF  CMU  VARIABLE  OF  CMR  VARIABLE 

IN  AIR  FORCE  AND  GENERAL  TEXT  SAMPLES  IN  AIR  FORCE  AND  GENERAL  TEXT  SAMPLES 
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FIGURE  8.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES  FIGURE  9.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES 

OF  DMU  VARIABLE  OF  YD  VARIABLE 

IN  AIR  FORCE  AND  GENERAL  TEXT  SAMPLES  IN  AIR  FORCE  AND  GFNERAL  TEXT  SAMPLES 
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FIGURE  12.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES  FIGURE  13.  PERCENTILES  CORRESPONDING  TO  RAW  SCORES 

OF  CE  VARIABLE  OF  LB  VARIABLE 

IN  AIR  FORCE  AND  GENERAL  TEXT  SAMPLES  IN  AIR  FORCE  AND  GENERAL  TEXT  SAMPLES 


The  plots  of  ('■Ml!  (Figure  3)  in  the  general  and  in  the  Air 
Force  materials  were  rather  similar,  although  the  Air  Force  ma- 
terials are  suggested  by  the  plots  to  be  slightly  more  difficult  (p 
.01,  sign  test). 

As  indicated  in  Figure  4,  the  two  samples  examined  were 
quite  similar  in  level  of  MMl  . The  Air  Force  materials  presented 
slightly  less  mental  load  (p  < . 01,  sign  test)  through  this  variable. 
This  relationship  is  consistent  with  that  of  CAIl'  and  might  be  ex- 
pected on  the  basis  of  the  similarity  of  the  measurement  formulas. 

The  ESI  plots  of  Figure  5 are  nearly  identical  with  some 
trend  towards  less  difficulty  for  the  Air  Force  materials  (p  < . 05, 
sign  test).  The  ESI  measure  reflects  the  frequency  of  occurrence 
of  abbreviations.  In  each  sample,  nearly  half  of  the  passages  were 
free  of  abbreviations.  The  most  difficult  (by  this  index)  general 
samples  appeared  to  show  a higher  frequency  of  abbreviations  than 
did  the  most  difficult  Air  Force  passages. 

The  general  sample  and  the  Air  Force  sample  were  about 
equivalent  in  terms  of  aids  to  comprehension  (N.VIS).  This  finding 
(p  >.  05,  sign  test)  (Figure  6)  was  not  anticipated  because  one  would 
expect  more  explanatory  information  in  passages  developed  for  train- 
ing purposes. 

XAII  considers  the  number  of  parts  of  speech  of  the  individual 
words  in  a passage.  As  indicated  in  Figure  7,  the  Air  Force  tech- 
nical writing  tended  to  be  easier  (p  < . 01,  sign  test)  than  the  general 
sample  on  this  index. 

The  Air  Force  and  the  general  samples  were  nearly  identical 
(p  >.  10,  sign  test)  in  terms  of  DMU  (a  measure  of  the  occurrence  of 
examples),  as  indicated  by  Figure  8.  Roughly  20  percent  of  the  pas- 
sages in  each  group  contained  examples  which  were  hypothesized  to 
aid  the  reader's  comprehension.  (A  mean  value  of  0.  0 may  be  ob- 
tained only  if  the  individual  values  averaged  are  equal  to  0.  0,  indicat- 
ing an  absence  of  examples  in  the  averaged  group.  ) 


Psycholi  nguistieallv  Oriented  Variables 


As  measured  by  Yngve  depth  (YD),  the  curves  for  the  two 
samples  crossed.  The  sign  test  indicated  no  statistically  signifi- 
cant difference  (p  >.05).  These  data  are  graphed  in  Figure  9. 

Figure  10  shows  the  comparable  plots  relative  to  the  mor- 
pheme depth  (\ID)  variable.  The  general  materials  seem  to  be 
easier  (p  < . 05,  sign  test)  in  this  regard. 

The  sample  of  general  materials  demonstrated  higher  (p  < 

. 01,  sign  test)  transformational  complexity  (TC)  scores  than  the 
comparative  Air  Force  sample.  Figure  11  indicated  that  one-third 
of  the  general  passages  were  entirely  composed  of  active  sentences. 
The  active  sentence  received  the  highest  transformational  complex- 
ity score.  (A  passage  can  only  receive  a TC  score  of  1.0  if  all  of 
its  sentences  are  active.  Similarly,  for  a group  mean  to  equal  1.  0, 
all  values  contributing  to  the  mean  must  equal  1.  0.  ) I. ess  than  10 
percent  of  the  Air  Force  samples  were  entirely  composed  of  active 
sentences.  Of  the  six  popular  passages  in  which  all  sentences  were 
active,  five  were  from  the  Reader's  Digest,  the  writings  of  Heming- 
way, and  the  writings  of  Steinbeck. 

The  Air  Force  materials  tended  to  be  about  equivalent  (p  > . 5, 
sign  test)  to  the  popular  samples  in  center  embedded  (CE)  phrases 
(Figure  12).  However,  the  proportions  of  each  type  of  passage  which 
were  free  of  center  embedded  material  were  equivalent.  (Mean  value 
of  1.  0 reflects  an  absence  of  center  embedded  material  in  the  aver- 
aged materials.  ) 

Approximately  two-thirds  of  general  passages  were  free  of 
left  branching  material  as  indicated  by  values  of  1.  0 for  the  33rd  and 
all  higher  percentiles.  Only  20  percent  of  the  Air  Force  passages 
were  similarly  unencumbered.  The  sign  test  indicated  statistical 
significance  at  the  . 01  level  of  confidence.  Additionally,  the  gen- 
eral passages  exhibiting  left  branching  material  did  so  to  a far 
lesser  degree  than  the  Air  Force  materials  (Figure  13).  f.eft 
branching  is  generally  due  to  clauses  which  modify  the  subject  of 
a sentence.  The  high  rate  of  occurrence  of  left  branching  in  the 
Air  Force  materials  may  reflect  the  preponde ranee  of  part  names 
with  multiple  modifiers  found  in  technical  materials. 
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As  indicated  by  Figure  14,  the  most  difficult  passages  in 
the  two  sets  were  equivalent  in  terms  of  right  branching,  but  the 
less  difficult  Air  Force  sample  passages  were  not  improved  in 
terms  of  right  branching  as  much  as  were  the  less  difficult  gen- 
eral passages.  That  is,  at  equivalent  percentile  values,  the  Air 
Force  sample  exhibited  a lower  1111  score  (p  .01,  sign  test) 
than  did  the  general  sample. 

As  shown  in  Figure  15,  a majority  of  both  groups  of  pas- 
sages was  found  free  of  deleted  complements.  The  sign  test  in- 
dicated no  statistically  significant  difference  between  the  two 
samples. 

In  addition,  duAlas'  coefficient  of  profile  similarity  (Mosel 
x Roberts,  1954)  was  computed  to  compare  the  shapes  of  the  dis- 
tributions of  the  14  variables  in  the  Air  Force  and  in  the  general 
sample.  The  value  profiled  was  the  percentile  score  of  each  dis- 
tribution which  corresponded  to  the  value  of  the  mean  of  the  values 
making  up  that  same  distribution. 

Profiles  composed  of  the  measures  described  were  con- 
structed for  the  Air  Force  sample  and  for  the  general  sample. 

The  variables  were  placed  in  the  order  in  which  they  were  defined 
in  the  present  chapter.  The  obtained  duAlas  coefficient  of  profile 
similarity  was  . 66.  Since  this  statistic  may  range  from  -1.  0 to 
+ 1. 0,  the  obtained  value  indicates  substantial  similarity  in  the  pro- 
files. Although  the  various  profiles  are  modestly  displaced  from 
each  other,  they  are  quite  similar  in  form;  the  primary  difference 
was  in  the  magnitude  of  the  raw  scores. 
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As  indicated  in  ( hapten  I,  prion  studies  have  resulted  in  the 
development  of  a set  of  psycholinguistic  and  SI  oriented  variables. 
However,  the  ac  tual  predictive  power  of  these  variables  has  not  been 
established  tv.  , variable's  singly  or  in  combination.  Hence,  the 
research  de  a <1  in  this  chapter  was  intended  to  generate  a com- 
p rehens ihi ! its  t [nation  that  could  be  used  to  predict  the  difficult}  of 
passages  of  textual  material. 

Subjects 


I w i > groups  of  subjects  were  involved.  One  group  (non  high 
school  - ite  1 ri  pri  sented  low  reading  ability  and  the  other  (col- 
lege stude  it  1 represented  high  reading  ability.  The  low  reading 
abili  t\  gi  ’ii'  w a c i:.i|)ci-rd  of  30  paid,  volunteer  subjects  who  were 
student tcch-i;  .il  job  training  program.  The  high  reading  abil- 
•it\  group  ua  on  pc  ed  of  2 1 similarly  paid  volunteer  college  under- 
graduate -tudellts. 

• ju  d st  r i won  of  the  Nelson -I  Jenny  Reading  Test  scores 
and  corresponding  reading  grade  levels  of  each  subject  group  is 
shown  in  figure  16.  Hie  total  group  is  divided  into  low-  and  high- 
reading  level  groups  at  the  10.  0 reading  grade  level.  The  low  read- 
ing level  group  contains  the  lower  scoring  26  of  the  subjects  obtained 
through  the  job  training  program.  The  high  reading  level  group  con- 
tains four  subjects  from  the  job  training  group  who  scored  above  the 
10.  0 grade  level  on  the  Nelson-Denny  test,  as  well  as  the  21  college 
undergraduates. 

Passage  Selection 

The  comprehensibility  equation  development  was  based  on  a 
set  of  passages  selected  from  Air  Force  Career  Development  Course 
((  DC)  texts.  ( DCs  are  uniquely  characterized  among  Air  force 
technical  publications  by  a predominance  of  connected  prose.  In 
( DCs,  textual  characteristics  such  as  outlines  and  numbered  lists 
of  procedural  steps  (which  are  quite  common  in  study  guides,  tech- 
nical orders,  and  manuals)  are  relatively  rare. 
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Fourteen  ( DC  text  passages  were  selected  from  three  sets 
of  texts.  These  text  sets  are  listed  in  Table  5. 


Table  5 

('DCs  Sampled  for  Predictive  Study 

a ••  l • 1 Air-craft  Ma irit-'iianoe  ’ ; c-oiali.  t , 

,■1  ii'cr-aft , One  and  Two  Engines 

43113  Aircraft  Mechanic 

645!  Inventory  Manag<  mi ;nt  Specialist 


The  passages  were  taken  at  equal  intervals  through  the  total 
number  of  column  pages  contained  in  the  three  sets  of  texts.  The 
procedure  followed  duplicated  that  employed  for  selection  of  the  Air 
Force  normative  sample,  described  in  Chapter  II.  Through  this  pro- 
cedure, three  samples  were  selected  from  ( DC  43151C,  five  from 
- ( DC  43113  and  six  from  CDC'  64550.  Each  passage  was  of  such 

length  as  to  contain  six  blocks  of  roughly  100  words  of  prose.  The 
word  count  for  each  block  began  at  the  beginning  of  a sentence  and 
continued  until  the  first  sentence  end  following  the  99th  successive 
word. 

Preparation  of  Stimulus  .Materials 

Cloze  test  forms  were  prepared  from  each  passage  by  delet- 
ing every  10th  word.  The  first  word  deleted  in  each  passage  was  a 
random  selection  from  the  first  ten  words  of  the  passage.  Cloze 
test  forms  were  typed  in  double-spaced  format.  No  deletions  were 
» made  to  subheads.  Figures,  diagrams,  or  tables  referenced  in  the 

passages  were  not  shown,  since  the  variables  under  investigation 
are  only  measured  in  connected  prose.  Test  booklets  were  then  as- 
sembled so  that  each  booklet  contained  all  14  passages  and  the  order 
of  appearance  of  passages  within  booklets  was  individually  random- 
,•  ized. 
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lavlor  (1953),  in  initiall\  presenting  the  cloze  procedure, 
reported  that  the  procedure  is  not  sensitive  to  the  scoring  criteria 
employed.  According'to  Taylor,  tin  resultant  ordering  of  passage 
scores  is  not  changed  if  synonyms  of  deleted  words  are  accepted 
as  correct  or  if  only  precise  matches  are  accepted.  In  the  current 
work,  a relatively  strict  scoring  procedure  was  followed; 


• only  precise  matches  of  deleted  words  or 
matches  involving  obvious  errors  of  spell- 
ing, tense,  or  number  were  accepted  as 
correct; 

• if  a number  was  required  (e.  g.  , a table  or 
figure  number,  model  number)  any  number, 
whether  entered  as  digits  or  written  out,  was 
accepted. 


In  all  cases,  cloze  test  scores  are  reported  as  percentage  of  de- 
leted words  correctly  entered,  according  to  the  criteria  described 
above.  'This  is  the  "normal"  cloze  score. 

Procedure 

Data  collection  required  a period  of  approximately  six  hours. 
All  subjects  were  initially  administered  the  Nelson-Denny  Heading 
Test  (Revised),  Form  A (Brown,  I960).  Instructions  were  then  pre- 
sented relative  to  the  procedures  to  be  followed  for  completing  the 
cloze  passages.  The  subjects  were  allowed  18  minutes  to  work  on 
each  passage.  They  were  instructed  to  proceed  to  a subsequent 
passage  only  at  the  completion  of  an  18-minute  time  block,  and  not 
to  go  back  to  an  earlier  passage  after  the  completion  of  its  work 
period.  A pilot  test  indicated  that  this  work  rate  allowed  suffici- 
ent time  for  subjects  to  complete  each  passage,  while  ensuring, 
that  subjects  di'd  not  dwell  excessively  on  difficult  items.  A 10- 
minute  break  was  taken  after  every  four  passages.  The  test  ses- 
sion for  the  job  training  group  subjects  was  additionally  broken  by 
a one-hour  lunch  break.  The  college  subjects  took  the  Nelson- 
Denny  test  and  six  cloze  passages  on  one  test  day,  and  the  remain- 
ing eight  cloze  passages  on  the  succeeding  day. 
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Predictor  Data 


The  level  of  each  of  the  psycholinguistic  and  the  SI  oriented 
variables  in  each  of  the  six  blocks  of  each  of  the  14  passages  repre- 
sented the  predictors  required.  These  variables  were  measured 
exactly  as  they  had  been  during  the  collection  of  normative  data, 
and  the  same  group  of  analysts  who  collected  the  normative  data 
performed  the  required  analyses.  The  percentile  levels  of  the  14 
variables,  as  shown  in  the  developed  norms,  were  tabulated  separ- 
ately for  each  block  of  each  passage,  and  then  averaged  over  the 
passage. 

Data  Analysis  and  Results 

A conventional  cross -validation  approach  was  used  to  accom- 
plish the  objective  noted  above.  Since  each  of  the  51  subjects  obtain- 
ed a cloze  score  on  each  of  14  passages,  the  data  were  regarded  as 
14  x 51  = 714  observations  to  be  regressed  on  the  14  independent  vari- 
ables (comprehensibility  measures).  Thus,  the  matrix  that  was 
used  in  the  multiple  regression  analysis  was  714  x 15.  However, 
since  this  matrix  contains  unwanted  subject  effects,  the  analysis 
was  preceded  by  partialing  out  these  effects.  This  was  accomplish- 
ed by  constructing  a design  matrix  representing  between  subject  ef- 
fects and  partialing  it  out  of  the  714  x 15  matrix.  The  resulting  ma- 
trix contains  variance  attributable  to  passages  only,  and  thus  is  ap- 
propriate for  the  analysis  described  below. 

The  subject  pool  was  randomly  divided  into  halves  and  a 
multiple  step-wise  regression  analysis  was  performed  on  the  data 
for  each  half  (designated  as  Hi  and  H2).  Each  equation  thus  de- 
rived was  then  applied  to  the  data  from  the  other  half  of  the  sam- 
ple and  the  multiple  Rs  were  observed  for  shrinkage.  The  valid- 
ities resulting  from  this  procedure  are  presented  in  Table  6.  Re- 
ferring to  the  table,  the  reader  can  see  that  the  multiple  R derived 
from  the  HI  data  dropped  by  only  . 083  when  the  equation  was  ap- 
plied to  the  H2  data;  and  that  derived  from  the  112  half  did  not  dimin- 
ish at  all  when  the  equation  was  applied  to  HI.  Since  the  shrinkage 
of  the  validities  was  negligible,  a regression  analysis  was  per- 
formed on  the  combined  data.  This  analysis  yielded  the  following 
equation: 
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in  which  each  of  the  comprehensibility  measure  values  is  in  the  form 
of  percentiles  taken  from  the  norm  tables  in  Appendix  A of  this  report. 
The  cloze  score  derived  from  this  equation  is  an  estimate  of  the  tra- 
ditional cloze  value;  i.  e.  , the  percentage  of  deletions  from  the  text 
that  are  correctly  filled  in. 


Table  6 

Validities  of  Equations  Developed  on  Both  Halves 
of  the  Sample  and  the  Total  Sample 


Total 

HI 

H2 

Total 

. 601 

HI 

.598 

.515 

H2 

. 599 

.592 

While  the  reader  who  is  interested  in  determining  the  com- 
prehensibility of  text  would  probably  use  the  above  equation,  it 
seemed  of  interest  to  compute  comparable  equations  for  each  of  the 
two  reading  ability  groups.  Table  7 presents  the  weights  for  each 
of  the  variables  of  the  equations  in  the  order  in  which  they  were 
added  to  the  regression  equation  and  the  constants  associated  with 
each  equation.  It  may  b‘e  noted  that  AT  ATE,  ESI,  and  YD  were  the 
first  three  factors  for  each  equation,  as  they  did  when  all  the  data 
were  combined.  Even  though  most  of  the  variance  is  accounted  for 
by  these  three  factors  a noteworthy  observation  is  the  occurrence 
of  only  two  common  factors  among  the  remaining  measures  in  the 
equations,  and  the  reversal  of  signs  for  MATE  and  AID  between  abil- 
ity groups.  This  may  mean  that  low  and  high  ability  groups  process 
information  differently  and  are  differentially  able  to  cope  with  the 
comprehensibility  factors.  Of  course,  these  differences  between 
high  and  low  groups  may  also  be  sample  specific  and  should  be  repli- 
cated. A complete  explanation  of  these  findings  awaits  further  re- 
search. 
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Table  7 


Regression  Equations  for  Low  and  High  HG1,  Groups 

Low  RGI.  Group  High  RGL  Group 


. 1 32 

MMU 

-.332 

MMU 

. 164 

ESI 

.171 

ESI 

. 200 

YD 

.418 

YD 

-.207 

CE 

.397 

TC 

.250 

RB 

.302 

CMU 

-.151 

MD 

. 089 

MD 

-.289 

NMI 

.320 

DMU 

-.074 

LB 

.167 

RB 

1 

o 

o 

CD 

[ Intercept ) 

-. 509 ( Intercept ) 

Discussion 


The  work  reported  in  Chapter  III  was  completed  in  order 
to:  (1)  provide  additional  verification  of  psycholinguisticali y and 
SI  oriented  variables  as  measures  of  textual  comprehensibility, 
and  (2)  provide  a method  for  predicting  the  comprehensibility  of 
textual  materials.  Both  of  these  goals,  established  at  the  outset, 
seem  to  have  been  achieved. 

In  this  regard,  we  note  that  the  materials  on  which  the 
current  effort  was  based  should  be  considered  in  the  evaluation 
of  the  multiple  correlation  for  the  regression  equation.  Predic- 
tions of  comprehensibility  have  customarily  been  established  us- 
ing written  materials  which  extend  across  a wide  range  of  reading 
difficulty,  such  as  Grades  3-12,  Grade  4 - College,  Grade  1 - Pro- 
fessional, etc.  The  materials  used  here  were  all  technical  ma- 
terials of  high  content  difficulty  and  intended  for  adult  readers. 
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Such  a restriction  of  range  in  the  criterion  would  be  expected  to 
attenuate  the  multiple  correlation.  Nonetheless,  the  obtained 
multiple  correlation  value  of  . 601  was  acceptable.  Accordingly 
increased  confidence,  in  the  value  of  the  variables  is  afforded  by 
these  results. 


IV.  DISCUSSION  AND  CONC  I 1SIONS 


The  present  work  was  performed  to  extend  and  verify  prior 
work  relative  to  a set  of  psycholinguistically  and  SI  oriented  vari- 
ables as  they  affect  the  comprehensibility  of  textual  materials.  In 
the  work  performed,  norms  and  a multiple  linear  regression  equa- 
tion relating  selected  psycholinguistically  and  SI  oriented  variables 
to  textual  comprehensibility  were  derived,  and  a conventional  cross- 
validation  was  performed. 

Traditionally,  the  developers  of  readability  equations  have 
warned  against  employing  such  equations  in  reverse  as  "rules  for 
writing.  " Throughout  the  development  and  investigation  of  the 
characteristics  of  the  psycholinguistic  and  the  SI  oriented  variables, 
it  has  been  held  that  the  present  variables  possess  diagnostic  and 
prescriptive,  as  well  as  descriptive,  value.  These  arguments 
seem  supported  on  the  bases  that  the  present  variables  consider 
the  linguistic  and  the  intellective  processes  involved  in  written 
language  decoding.  There  was  little,  if  anything,  in  the  present 
work  to  negate  the  contentions  of  the  prior  work  relative  to  the  val- 
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hensibility measurement,  prediction,  and  improvement. 
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APPENDIX 


Tables  of  Norms 


To  aid  the  reader  In  Interpreting  th< 
tables  presented  in  this  appendix  in 
example  of  how  Table  A-l  should  to  read 
i ollows : ten  percent  ' th<  t idy  gui  l< 
samples  had  a CMU  value  of  It  . than 
.288;  over  all  of  th<  material  ten  per- 
cent of  the  samples  Lad  a IT'  value  of 
less  than  *293,  Ninety  pei  ent  >f 
study  guide  samples  had  a CMU  val  -■  >1 

less  than  .470;  over  all  the  materials, 
ninety  percent  ■:  th<  imj  L<  had  a CM 
value  of  less  than  .476. 
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Cognition  of  Semantic  Units  (CMU)  Cognition  of  Semantic  Relations  (CMR) 
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[See  page  49  for  sample  interpretation  of  these  tables. 


Convergent  Production  of  Semantic  Systems  (NMS)  Convergent  Production  of  Semantic  Implications  (NMI) 
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[See  page  49  for  sample  interpretation  of  these  tables. 


Table  A -9  Table 
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[See  page  49  for  sample  interpretation  of  these  tables. 


Right  Branching  (RB)  Deleted  Complement  (DC) 
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[See  page  49  for  sample  interpretation  of  these  tables. 


Table  A- 15 
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